METHOD OF IMAGE BACKGROUND REPLACEMENT 
BACKGROUND OF THE INVENTION 

1. Field of the Invention 

This invention relates to image processing for image capture systems, 
more particularly to replacing backgrounds in captured images. 

2. Background of the Invention 

New applications for video technology are appearing every day. Video 
conferencing has become commonplace, users can easily download video clips 
from the Internet, and camcorders are inexpensive and widely used. As more of 
these applications develop, the ability to edit and change the nature of the 
images becomes more important. 

One editing tool that remains unavailable to most of these applications is 
background replacement. Typically, background replacement occurs on video 
images filmed in front of a blue screen. The foreground and background pixels 
are easily identified and the new background is put in place using computers or 
specially designed video devices. However, for most people, filming or capturing 
images in front of a blue screen is impractical. 

The blue screen process is expensive and inconvenient. A special studio 

for video conferencing restricts the availability of video conferencing facilities and 

requires extra cost. Most people publishing on the Internet would find use of a 

blue screen prohibitive, as would most typical users of camcorders. However, all 

of these applications can benefit from background replacement. People video 

conferencing could replace the background of their office with a background for 
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reasons of privacy, security or aesthetics. Internet publishers could insert 
* images into Web pages more seamlessly, without use of backgrounds or sets. 
Camcorder users could record videos and edit the backgrounds at home. 

Therefore, a less expensive and more easily accessible technique for 
background replacement is needed. 
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SUMMARY OF THE INVENTION 

One embodiment of the invention is a technique for background 
replacement. The input image or images are analyzed and a preliminary 
classification of the pixels is made. The classification identifies whether the 
pixels are more likely foreground or background. After the preliminary 
classification is made, a more refined process is applied that makes the final 
determination. Finally, the new background pixels are applied to the image, 
replacing the previous background pixels. The new image is composed with 
feathering to ensure smooth edges and transitions. The new image is then 
output for viewing. 

It is an advantage of the invention in that it allows background 
replacement with no extra equipment or special settings. 

It is an advantage of the invention in that it provides background 
replacement quickly, allowing real-time processing. 

It is an advantage of the invention in that it is able to adjust for camera 
exposure changes and accurate in determining background pixels from 
foreground pixels. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the present invention and for 
further advantages thereof, reference is now made to the following Detailed 
Description taken in conjunction with the accompanying Drawings in which: 

Figure 1 shows a process for video background replacement in 
accordance with the invention. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

One embodiment of a process for video background replacement is 
shown in Figure 1. An input device in 10 is used to capture images. Devices 
such as these include digital cameras, camcorders, film cameras, video 
conferencing cameras, etc. In step 10 the current background is recorded 
without any foreground object or objects. 

Ttae device 10 then takes incoming frames or an incoming frame of the 
image withNhe foreground obejcts as the input image in step 12. For digital 
cameras, the iKput image would be that one image captured by the image input 
device. The input aevices that are video images may capture one or more 
frames to use as the irWjt image or in the input image analysis. 

The input image is then analyzed using a probability function that 
measures the likelihood of the pixel being foreground or background. One 
example of such a probability function is: 

P(p M , e Foreground) = H« ' ^ " ^ ^ + ~ f + * ~ r »^ > « 

[0(d- 1 I x , y -I' x , y \+f) else 

and 

O(w) = min(max(0.5 + sign(u) -u , 0), 1) 

where r and g are the chromatic components and I is the intensity of the pixel p; 
r', g' and I' are their counter parts of pixel p' in the pre-recorded background 
image, and a, b, c, d, I and rj are constants. The values of these constants are 
tuned by experiments to determine their optimal values. 
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It is not necessary to restrict this process to chromatic or normalized RGB 
color space. Use of YC b C r is also possible. In the YC b C r example, the same 
formulas would be used, with the substitutions of Y for I, C b for g and C r for r. 

Regardless of the color space used for determining the preliminary 
classification of a pixel, a probability map is generated that indicates the 
likelihood of a pixel being foreground or background. The probability map 
produces a value of a pixel between 0 and 1 , where 0 is the foreground and 1 is 
the background in this particular example. These probabilities could have a 
threshold applied that would segment the pixels into either the foreground or 
background. However, this may lead to false classifications because of 
ambiguity in certain regions in foreground objects and the background. 

Therefore, it is desirable to refine the classification result by utilizing 
certain context information in space. One may apply morphological filtering to 
eliminate isolated mis-classified pixels. Other techniques are also available for 
this post-processing refinement in step 16. One such technique is anisotropic 
diffusion, which is discussed below. 

Anisotropic diffusion encourages smoothing within boundaries and 
discourages smoothing across boundaries. In this example, the following 
anisotropic diffusion equation will be used: 

P t = div(c(x,y,t)VP) = c(x,y,t)AP + Vc • VP 9 

where div denotes the divergence operator, and v and a denote the gradient 
and Laplacian operators, respectively, with respect to the space variables. The 
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continuous diffusion equation may be discretized on a square lattice. Using a 4- 
nearest-neighbors discretization of the Laplacian operator, the equation 
becomes: 



= K,y + Xlc n ■V n P + c s -V s P + c e -V e P + c w -V w P]' Xk 

and 



VP = P — P 

V N 1 x,y 1 x,y-\ 1 x,y 

VP — P — P 

v S 1 x,y 1 x 9 y+\ 1 x,y 

V P - P -P 

2? x,y x+l ?< y x 9 y 

V P —P —P 

W x,y x-\y xy 

where o<a<i/4 for numeric stability reason, A/, S, E, W denote North, South, 
East and West, respectively. The conduction coefficients c Nl c s , c Ef c w may be 
computed as follows: 

% =*flV v |) 

and 

1 



^(|V/|): 



1 + (|V/|/7Q Z 
where K is a constant, e.g., K = 1000. 

This refined probability map from step 16 is then used to overlay 
foreground pixels on a new background. Some type of blending or feathering 
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process should be used. Feathering as used here denotes any kind of process 
that does not just overlay the pixels with no comparison whatsoever between the 
foreground and background. Specifically, in this example, a weighted average 
over the pixel value of the input image and the pixel value of the new 
background is applied. The weights are determined by the probability value from 
the probability map. 

The example of this feathering algorithm for a given location (x,y) in the 
output image, has the following formulas: 

K7 Ut = P(p x , y e Foreground) ■ R^ 1 + (1 - />(p^ s Foreground)) ■ r™*"*"** 
G° x « put = P(p x , y e Foreground) • + (1 - P(p x , y e Foreground)) ■ G x ^ backsrou,td 
KT* = P (V*,y G foreground) ■ + (1 - P{ Vx y e Foreground)) • B ™J back8 ™" d . 

Once the feathering is complete in step 18, the output image with the new 
background is produced. While the input may be a video image, this technique 
can be used for printed output as well, such as paper, postcards, photographic 
paper, etc. 

Several modifications of this process are possible. As mentioned 
previously, the above example relies upon RGB color space for discussion 
purposes. Other types of processing, including YC b C r , can be used. The 
selection of the number of frames used is also left up to the designer. It is 
possible that several frames could be analyzed with associated motion analysis 
as well, to ensure the highest accuracy of the fore/back ground classification. 
The use of the nearest neighbor is not limited to four neighbors. The selection of 
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these specifics is left to the designer based upon the computational power of the 
system and the requirements of the final image. 

Similarly, while the above process relies upon anisotropic diffusion for the 
refinement of classification, other types of refinements are available, such as 
morphological filtering, as mentioned above. 

Application of this invention results in several options for users. A video 
conference participant can shield the actual background of the room from those 
at the receiving end of the image, for privacy or security reasons. 

A Web publisher can generate transparent images in GIF format much 
more quickly than presently possible. Current techniques involve a pixel by pixel 
designation by the user to identify foreground and background pixels, a 
painstaking and tedious process. These same techniques are required when 
consumers using digital cameras want to crop and move objects in their digital 
images, whether video or still. These problems are eliminated by the application 
of this invention. 

Thus, although there has been described to this point a particular 
embodiment for a method to perform background replacement, it is not intended 
that such specific references be considered as limitations upon the scope of this 
invention except in-so-far as set forth in the following claims. 



SLA 0095 Page 9 



